Lane fusion system using forward-view and rear-view cameras

ABSTRACT

A method and system for computing lane curvature and a host vehicle&#39;s position and orientation relative to lane boundaries, using image data from forward-view and rear-view cameras and vehicle dynamics sensors as input. A host vehicle includes cameras at the front and rear, which can be used to detect lane boundaries such as curbs and lane stripes, among other purposes. The host vehicle also includes vehicle dynamics sensors including vehicle speed and yaw rate. A method is developed which computes lane curvature and the host vehicle&#39;s position relative to a lane reference path, where the lane reference path is derived from the lane boundaries extracted from a fusion of the front and rear camera images. Mathematical models provided in the disclosure include a Kalman filter tracking routine and a particle filter tracking routine.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates generally to a lane position method and systemfor a vehicle and, more particularly, to a lane position method andsystem for a host vehicle which uses a fusion of image data fromforward-view and rear-view cameras and data from vehicle dynamicssensors to compute lane parameters, including lane curvature and thehost vehicle's position and orientation relative to a lane referencepath.

2. Discussion of the Related Art

Many modern vehicles include onboard cameras, which are used for avariety of purposes. One common application is a forward-viewing camerawhich can provide images to be used in a collision avoidance system, alane departure warning system, a lateral control system, or acombination of these or other systems. However, conditions may arisewhich prevent a good image from being obtained from the forward-viewingcamera. Such conditions include a leading vehicle at close range whichblocks much of the camera's field of view, and low-visibility weatherconditions, such as rain and fog, which obscure the camera's image. Insuch conditions, when a usable image from the forward-view camera is notavailable, systems which rely on the camera's image for input cannot beoperated.

Meanwhile, many newer vehicles are also equipped with a rear-viewcamera, which is normally used only for backup assistance, such asproviding a video image for the driver to see what is behind thevehicle. Although these rear-view cameras typically have a resolutionand field of view which are more than sufficient for other image datacollection purposes, until now they have not been used to supplement theimages from forward-view cameras for lane position and lateral controlapplications.

There is an opportunity to use the image data available from a rear-viewcamera, and combine it with image data from a forward-view camera andother sensors, to provide a more robust lane sensing system. Theresultant two-camera system not only makes use of more input data undernormal conditions, but also provides a usable source of image data toallow operation of the system when conditions are unfavorable forforward-view imaging.

SUMMARY OF THE INVENTION

In accordance with the teachings of the present invention, a method andsystem are disclosed for computing lane curvature and a host vehicle'sposition and orientation relative to lane boundaries, using image datafrom forward-view and rear-view cameras and vehicle dynamics sensors asinput. A host vehicle includes cameras at the front and rear, which canbe used to detect lane boundaries such as curbs and lane stripes, amongother purposes. The host vehicle also includes vehicle dynamics sensorsincluding vehicle speed and yaw rate. A method is developed whichcomputes lane curvature and the host vehicle's position relative to alane reference path, where the lane reference path is derived from thelane boundaries extracted from a fusion of the front and rear cameraimages. Mathematical models provided in the disclosure include a Kalmanfilter tracking routine and a particle filter tracking routine.

Additional features of the present invention will become apparent fromthe following description and appended claims, taken in conjunction withthe accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a vehicle lateral control system which usesfront and rear cameras and other sources of input;

FIG. 2 is a diagram of a bicycle model for lateral control of a hostvehicle;

FIG. 3 is a diagram of the host vehicle showing many of the keyparameters of the lateral control model;

FIG. 4 is a control block diagram showing how the vehicle lateralcontrol model is implemented;

FIG. 5 is a block diagram of a system for vehicle lateral control usinga 2-camera lane fusion approach;

FIG. 6 is a block diagram of a first embodiment of a lane fusion systemusing input from two cameras;

FIG. 7 is a block diagram of a second embodiment of a lane fusion systemusing input from two cameras;

FIG. 8 is a diagram which shows an example of lane stripe representationfor a scenario where several short stripes and one long arc have beendetected;

FIG. 9 is a histogram which shows how the displacement of the hostvehicle to the lane boundaries can be computed;

FIG. 10 is a flow chart diagram of the Kalman filter tracking methodused in the lane tracking module of FIG. 7; and

FIG. 11 is a flow chart diagram of the particle filter tracking methodused in the lane tracking module of FIG. 7.

DETAILED DESCRIPTION OF THE EMBODIMENTS

The following discussion of the embodiments of the invention directed toa lane fusion system using forward-view and rear-view cameras is merelyexemplary in nature, and is in no way intended to limit the invention orits applications or uses.

Many modern vehicles include forward-view cameras, and systems which usethe image data from the forward-view cameras in applications such aslane departure warning and lateral control assistance. However, imagesfrom forward-view cameras can be obstructed by a leading vehicle, orobscured by sun glare, fog, rain, or snow, which reduces the reliabilityof applications which would rely on the images. Given the increasingavailability of rear-view cameras, often used primarily for backupassistance, it makes sense to use the rear-view camera image data as asupplement to the forward-view camera image data. Along with GPS anddigital map data, vehicle dynamics sensors, and radar-based or othersystems capable of detecting a vehicle on the road ahead of the hostvehicle, the forward-view and rear-view camera image data can be used inadvanced applications for improved safety and vehicle control.

In one approach, the data sources are used directly in a vehicle lateralcontrol application. FIG. 1 is a block diagram of a system 10 forlateral control of a vehicle using forward-view and rear-view camerasand other data sources. The system 10 uses image data from aforward-view camera 12 and a rear-view camera 14, as will be discussedbelow. A leading vehicle position system 16, which may be a long rangeradar (LRR) or other type system, tracks the position of a leadingvehicle, for the purpose of estimating the path of the roadway. Roadcurvature information from a GPS-based navigation system or digital map18 provides another source of data for the system 10. The inputs fromthe forward-view camera 12, the rear-view camera 14, the leading vehicleposition system 16, and the digital map 18 are all used by a vehiclelateral control module 20, the operation of which will be discussed indetail below.

FIG. 2 is a diagram of a bicycle model 30 for vehicle lateral control,which is obtained by combining the two wheels of each axle into onewheel at the centerline of the vehicle. FIG. 3 is a diagram of a controlmodel 40 which adds more detail to the bicycle model 30. Like elementsand dimensions share the same reference numerals in FIGS. 2 and 3, whichwill be discussed together. The following table is provided as an indexof the items and dimensions shown in FIGS. 2 and 3, including theirreference numbers and descriptions.

Ref # Symbol Description 30 n/a bicycle model 40 n/a control model 50n/a host vehicle 52 n/a front tire 54 n/a rear tire 56 n/a center ofgravity point 60 n/a lane reference path 62 κ lane curvature 64 Δy_(F)front lateral displacement 66 Δy_(T) rear lateral displacement 68 d_(F)longitudinal distance in front of center of gravity 70 d_(T)longitudinal distance behind center of gravity 72 l_(F) distance offront axle from center of gravity 74 l_(T) distance of rear axle fromcenter of gravity 80 n/a leading target vehicle 82 X_(O) forwarddistance of leading target vehicle from center of gravity of hostvehicle 84 Y_(O) lateral distance of leading target vehicle from centerof gravity of host vehicle 86 θ_(O) heading angle of leading targetvehicle with respect to host vehicle 92 ν_(yH) host vehicle lateralvelocity 94 ν_(xH) host vehicle longitudinal velocity 96 ω_(H) hostvehicle yaw rate 98 δ_(F) front wheel steering angle 100 n/a vehiclepath 102 n/a heading line 104 α_(O) lateral offset between the headingline and the vehicle path at forward distance 82 106 ε_(O) lateraloffset between the vehicle path and the lane reference path at forwarddistance 82 108 α_(F) lateral offset between the heading line and thevehicle path at forward distance 68 110 ε_(F) lateral offset between thevehicle path and the lane reference path at forward distance 68 112α_(T) lateral offset between the heading line and the vehicle path atrearward distance 70 114 ε_(T) lateral offset between the vehicle pathand the lane reference path at rearward distance 70 120 θ_(F) vehicleorientation angle with respect to tangent to lane reference path atforward distance 68 122 θ_(T) vehicle orientation angle with respect totangent to lane reference path at rearward distance 70

A host vehicle 50 is the subject of the bicycle model 30 and the controlmodel 40, used in the vehicle lateral control module 20. The hostvehicle 50 is represented by a front tire 52, a rear tire 54, and acenter of gravity point 56 in the bicycle model 30. The host vehicle 50is assumed to be equipped with a yaw rate sensor (not shown), and othersensors as necessary to know its longitudinal and lateral velocity.

A lane reference path 60 is assumed to be the centerline of a circularlane path with curvature κ, an estimate of which comes from the digitalmap 18. For the augmented lateral control system as considered in thebicycle model 30, the lateral displacement of the host vehicle 50 fromthe lane reference path 60 is measured both as a front lateraldisplacement Δy_(F) and a tail lateral displacement Δy_(T) by theforward-view camera 12 and the rear-view camera 14, respectively. Thedisplacement measurements are acquired by the cameras at a longitudinaldistance d_(F) in front of the center of gravity point 56 and a distanced_(T) behind the center of gravity point 56. The distances d_(F) andd_(T) are time variant and dependent on the quality of lane markersdetected by the cameras 12 and 14, occlusion by leading or followingvehicles, and lighting conditions.

The leading vehicle position system 16 onboard the host vehicle 50 candetect a leading target vehicle 80, and provide its longitudinaldistance X_(O), lateral distance Y_(O), and heading angle θ_(O). Only avehicle immediately in front of the host vehicle 50 and within adistance threshold (e.g., 50 m) is considered as the leading targetvehicle 80. Other vehicle parameters in the bicycle model 30 aredistances l_(F) and l_(T) of the front and rear axles, respectively,from the center of gravity point 56. Three host vehicle state variablesare also shown: vehicle lateral velocity v_(yH) vehicle longitudinalvelocity v_(xH), and vehicle yaw rate ω_(H). A front wheel steeringangle δ_(F) is the input of the automatic steering system as commandedby the lateral control system 20.

A vehicle path 100 describes the path the host vehicle 50 is currentlyfollowing, and a heading line 102 represents a straight line through thecenterline of the host vehicle 50. Distance α_(O) is the lateral offsetbetween the heading line 102 and the vehicle path 100 at the forwarddistance X_(O). Distance ε_(O) is the lateral offset between the vehiclepath 100 and the lane reference path 60 at the forward distance X_(O).Distance α_(F) is the lateral offset between the heading line 102 andthe vehicle path 100 at the forward distance d_(F). Distance ε_(F) isthe lateral offset between the vehicle path 100 and the lane referencepath 60 at the forward distance d_(F). Distance α_(T) is the lateraloffset between the heading line 102 and the vehicle path 100 at therearward distance d_(T). Distance ε_(T) is the lateral offset betweenthe vehicle path 100 and the lane reference path 60 at the rearwarddistance of d_(T).

Vehicle orientation with respect to the lane reference path tangent atthe forward distance d_(F) is represented by angle θ_(F), and vehicleorientation with respect to the lane reference path tangent at therearward distance d_(T) is represented by angle θ_(T).

In addition to the elements and dimensions shown in the bicycle model 30and the control model 40, the following symbols must also be defined:m=Total mass of the host vehicle 50; I_(ω)=Total inertia of the hostvehicle 50 around the center of gravity point 56; l=Distance between thefront and rear axles, (l=l_(F)+l_(T)); and c_(F), c_(T)=Corneringstiffness of the front and rear tires, 52 and 54, respectively.

A linearized bicycle state-space model of the lateral vehicle dynamicscan be written as:

$\begin{matrix}{\begin{bmatrix}{\overset{.}{v}}_{yH} \\{\overset{.}{\omega}}_{H}\end{bmatrix} = {{\begin{bmatrix}{- \frac{c_{F} + c_{T}}{{mv}_{xH}}} & {\frac{{c_{T}l_{T}} - {c_{F}l_{F}}}{{mv}_{xH}} - v_{xH}} \\\frac{{{- l_{F}}c_{F}} + {l_{T}c_{T}}}{I_{\omega}v_{xH}} & {- \frac{{l_{F}^{2}c_{F}} + {l_{T}^{2}c_{T}}}{I_{\omega}v_{xH}}}\end{bmatrix}\begin{bmatrix}v_{yH} \\\omega_{H}\end{bmatrix}} + {\begin{bmatrix}\frac{c_{F}}{m} \\\frac{l_{F}c_{F}}{I_{\omega}}\end{bmatrix}\delta_{F}}}} & (1)\end{matrix}$

The state-space equations capturing the evolution of the forward-viewcamera measurements due to the motion of the host vehicle 50 and changesin the road geometry are:Δ{dot over (y)} _(F) =v _(xH)θ_(F) −v _(yH)=ω_(H) d _(F)  (2){dot over (θ)}_(F) =v _(xH)κ−ω_(H)  (3)

Similarly, the state-space equations capturing the evolution of therear-view camera measurements due to the motion of the host vehicle 50and changes in the road geometry are:Δ{dot over (y)} _(T) =v _(xH)θ_(T) −v _(yH)+ω_(H) d _(T)  (4){dot over (θ)}_(T) =v _(xH)κ−ω_(H)  (5)

It is assumed that the leading target vehicle 80 is following thecenterline of the lane reference path 60, thus the state-space equationscapturing the evolution of the radar measurements due to the motion ofthe host vehicle 50 and changes in the road geometry are:{dot over (Y)} _(O) =v _(xH)θ_(O) −v _(yH)−ω_(H) X _(O)  (6){dot over (θ)}_(O) =v _(xH)κ−ω_(H)  (7)

The vehicle lateral dynamics, front camera dynamics, rear cameradynamics, and leading target vehicle dynamics described in Equations(1)-(7) can then be combined into a single dynamic system of the form:

$\begin{bmatrix}{\overset{.}{v}}_{yH} \\{\overset{.}{\omega}}_{H} \\{\Delta\;{\overset{.}{y}}_{F}} \\{\overset{.}{\theta}}_{F} \\{\Delta\;{\overset{.}{y}}_{T}} \\{\overset{.}{\theta}}_{T} \\{\overset{.}{Y}}_{O} \\{\overset{.}{\theta}}_{O}\end{bmatrix} = {\begin{bmatrix}{- \frac{c_{F} + c_{T}}{{mv}_{xH}}} & {\frac{{c_{T}l_{T}} - {c_{F}l_{F}}}{{mv}_{xH}} - v_{xH}} & 0 & 0 & 0 & 0 & 0 & 0 \\\frac{{{- l_{F}}c_{F}} + {l_{T}c_{T}}}{I_{\omega}v_{xH}} & {- \frac{{l_{F}^{2}c_{F}} + {l_{T}^{2}c_{T}}}{I_{\omega}v_{xH}}} & 0 & 0 & 0 & 0 & 0 & 0 \\{- 1} & {- d_{F}} & 0 & v_{xH} & 0 & 0 & 0 & 0 \\0 & {- 1} & 0 & 0 & 0 & 0 & 0 & 0 \\{- 1} & d_{T} & 0 & 0 & 0 & v_{xH} & 0 & 0 \\0 & {- 1} & 0 & 0 & 0 & 0 & 0 & 0 \\{- 1} & {- X_{O}} & 0 & 0 & 0 & 0 & 0 & v_{xH} \\0 & {- 1} & 0 & 0 & 0 & 0 & 0 & 0\end{bmatrix}{\quad{\begin{bmatrix}v_{yH} \\\omega_{H} \\{\Delta\; y_{F}} \\\theta_{F} \\{\Delta\; y_{T}} \\\theta_{T} \\Y_{O} \\\theta_{O}\end{bmatrix} + {\begin{bmatrix}\frac{c_{F}}{m} \\\frac{l_{F}c_{F}}{I_{\omega}} \\0 \\0 \\0 \\0 \\0 \\0\end{bmatrix}\delta_{F}} + \begin{bmatrix}0 \\0 \\0 \\{v_{xH}\kappa} \\0 \\{v_{xH}\kappa} \\0 \\{v_{xH}\kappa}\end{bmatrix}}}}$or in short as:{dot over (x)}=f(x)+g(δ_(F))  (8)

Let y=[{dot over (ω)}_(H) Δ{dot over (y)}_(F) {dot over (θ)}_(F) Δ{dotover (y)}_(T) {dot over (θ)}_(T) {dot over (Y)}_(O) {dot over(θ)}_(O)]^(T) denote the output of the dynamic system, observed by theyaw rate sensor, the forward-view camera 12, the rear-view camera 14,and the leading vehicle position system 16. The observation equation canbe written as y=o(x).

Referring to the lane reference path 60 and the vehicle path 100 of FIG.3, the goal of the lateral control module 20 is to track the roadway byregulating the lateral differences between the lane reference path 60(i.e., Δy_(F), Δy_(T) and Y_(O)) and the vehicle path 100 (i.e., α_(F),α_(T), and α_(O)) at distances of d_(F), d_(T), and X_(O), measured bythe forward-view camera 12, the rear-view camera 14, and the leadingvehicle position system 16, respectively. Namely, the control objectiveis to minimize:J=w _(F)ε_(F) −w _(T)ε_(T) +w _(O)ε_(O)  (9)where ε_(F)=Δy_(F)−α_(F), ε_(T)=Δy_(T)−α_(T), and ε_(O)=Y_(O)−α_(O); andw_(F), w_(T), and w_(O) are normalized positive weights such thatw_(F)+w_(T)+w_(O)=1.

Equation (9) can then be written as:J=h(x)  (10)

Feedback linearization is a common approach used in controllingnonlinear system. The approach involves coming up with a transformationof the nonlinear system into an equivalent linear system through achange of variables and a suitable control input. The application ofthis technique to the bicycle model 30 is not linearization since thebicycle model 30 is already linear. But this technique can be applied torender the bicycle model 30 independent of the host vehicle longitudinalvelocity v_(xH).

The control law required to linearize the system expressed in Equations(8) and (10) by differentiating Equation (10) twice with respect to timeis as follows:

$\begin{matrix}{\delta_{F} = {\frac{1}{L_{g}L_{f}^{2}{h(x)}}\left( {{{- L_{f}^{2}}{h(x)}} + u} \right)}} & (11)\end{matrix}$where L^(i) _(ƒ) denotes the i-th Lie derivative along function ƒ. A Liederivative evaluates the change of one vector field along the flow ofanother vector field, as is known to those skilled in the art ofmathematics.

Employing this control law yields a second order equation of the form{umlaut over (J)}=u. Let z₁=J. The resulting simplified dynamic systemcan be expressed as:ż ₁ =z ₂ż ₂ =u  (12)

Using the following state feedback control law:u=−k ₁ z ₁ −k ₂ z ₂  (13)the second order system Equation (12) can be written as ż=Az, with

$A = {\begin{bmatrix}0 & 1 \\k_{1} & k_{2}\end{bmatrix}.}$

Therefore, with appropriate choice of k₁ and k₂, a stable lane trackingsystem can be designed with the eigenvector of A in the open left halfof the complex plane.

As shown in FIG. 1, the digital map 18 provides input to the lateralcontrol module 20, including an estimate of the lane curvature κ, whichcan be used as part of a feed-forward control strategy. By letting [{dotover (v)}_(yH) {dot over (ω)}_(H) Δ{dot over (y)}_(F) {dot over(θ)}_(F)]^(T)=0, the steering input, δ_(fwd), that tracks a lanecurvature κ can be computed from Equations (1)-(3) as:

$\begin{matrix}{\delta_{fwd} = {\kappa\left( {l - \frac{\left( {{l_{F}c_{F}} - {l_{T}c_{T}}} \right)v_{xH}^{2}m}{c_{T}c_{F}l}} \right)}} & (14)\end{matrix}$

This feed-forward component of Equation (14) can be added to the abovederived control law in Equations (11) and (13) to improve the transientbehavior of the host vehicle 50 when entering and exiting curves.

FIG. 4 is a control block diagram 140 which shows how the vehiclelateral control strategy described above is implemented. The steps inthe control method are outlined as follows:

-   -   1) At box 142, the digital map 18 provides an estimate of the        lane curvature κ on line 152.    -   2) At box 144, vehicle dynamics sensors provide the vehicle        forward velocity v_(xH) and yaw rate ω_(H) measurements on line        154.    -   3) At box 146, the forward-view camera 12 provides measurements        of the lane orientation θ_(F), the lateral displacement Δy_(F),        and the longitudinal distance where the measurement is taken        d_(F), on line 156.    -   4) At box 148, the rear-view camera 14 provides measurements of        the lane orientation θ_(T), the lateral displacement Δy_(T), and        the longitudinal distance where the measurement is taken d_(T),        on line 158.    -   5) At box 150, the leading vehicle position system 16 provides        leading target vehicle position, i.e., the longitudinal offset        X_(O), the lateral offset Y_(O), and the heading θ_(O), on line        160.    -   6) The inputs on the lines 152-160 are provided to box 170,        where the feed-forward term δ_(fwd) is computed as in Equation        (14).

7) At box 172, the feedback linearization term δ_(F), is computed as inEquation (11).

-   -   8) At summing junction 174, the feed-forward term δ_(fwd) and        the feedback linearization term δ_(F), are added together, and        sent to a steering actuator (electric power steering, or other        type system) in the host vehicle 50 at box 176.    -   9) At box 178, an Observer module estimates the vehicle's state        variables using a Kalman filter, with the data on the lines        152-160 and the vehicle's response as inputs, using Equation (8)        and y=o(x).    -   10) At box 180, a variable change module computes z₁ and z₂        using Equations (10) and (12).    -   11) At box 182, the feedback term u is computed for the        linearized dynamic system using Equation (12).

Some examples are provided to further explain the operation of thecontrol method described above. In the best case scenario, measurementsfrom all three external sensors are available; that is, rearward laneboundary information from the rear-view camera 14, forward lane boundaryinformation from the forward-view camera 12, and leading vehicleinformation from the leading vehicle position system 16. In such a case,the weight parameters in Equation (9) are defined to be proportional tothe quality of the measurement (i.e., signal-noise ratio, or variance ofthe estimates) returned by the corresponding sensors. For example, letmeasurement variances of the forward-view camera 12, the rear-viewcamera 14, and the leading vehicle position system 16 be σ_(F), σ_(T),and σ_(O), respectively. Then the corresponding weights are computed as:

$\begin{matrix}{{w_{F} = {C\;{\mathbb{e}}^{- \frac{\sigma_{F}^{2}}{W}}}},{w_{T} = {C\;{\mathbb{e}}^{- \frac{\sigma_{T}^{2}}{W}}}},{w_{O} = {C\;{\mathbb{e}}^{- \frac{\sigma_{O}^{2}}{W}}}}} & (15)\end{matrix}$where C is the normalization parameter such that w_(F)+w_(T)+w_(O)=1,and W is a bandwidth parameter chosen by the designer.

In a situation where the leading target vehicle 80 blocks the view ofthe forward-view camera 12, such that little or no forward lane boundaryinformation is available, the weight parameters of Equation (9) would betuned by decreasing the value of w_(F) (possibly to zero), andincreasing the values of w_(T) and w_(O). Similarly, in a situationwhere there is no suitable leading target vehicle 80, the value of w_(O)would be set to zero, and the values of w_(F) and w_(T) would beincreased. Finally, in a situation where a low-angle sun or inclementweather obscures the image from the forward-view camera 12, such that noforward lane boundary information is available, the weight parameters ofEquation (9) would be tuned by setting the value of w_(F) to zero, andincreasing the values of w_(T) and w_(O).

Using the control method described above, a robust vehicle lateralcontrol system can be implemented. By directly using front and rearcamera images as input, along with other indicators of road curvature,the lateral control system can provide more reliable and stableperformance than lateral control systems which do not use as manysources of input.

Another approach to vehicular lateral control can be achieved by firstcombining the data from the forward-view camera 12 and the rear-viewcamera 14 in a data fusion module, and then using the resultant lanecurvature and displacement information from the fusion module in alateral control module.

FIG. 5 is a block diagram of a system 200 for vehicle lateral controlusing a 2-camera lane fusion approach. Like the system 10 shown in FIG.1, the system 200 uses data from the forward-view camera 12, therear-view camera 14, the leading vehicle position system 16, and thedigital map 18. However, unlike the system 10 which uses the inputsdirectly in the lateral control module 20, the system 200 first combinesthe inputs in a data fusion module 210. The outputs of the data fusionmodule 210, including roadway curvature, and the vehicle's displacementand orientation relative to lane boundaries, are then provided to avehicle lateral control module 220. The outputs of the data fusionmodule 210 may also be used in applications other than a lateral controlsystem, such as a lane departure warning system.

Two methods of performing lane data fusion will be discussed below. Inthis discussion, many of the variables and dimensions from FIGS. 2 and 3will be referenced.

A traditional lane information system with lane departure warningtypically includes the forward-view camera 12 that can measure thevehicle orientation with respect to the tangent of the lane θ_(F) at thefront, the front lateral displacement Δy_(F) at the front bumper, andthe lane curvature κ, where the distance d_(F) is defined as thedistance from the center of gravity point 56 to the front bumper of thehost vehicle 50. Besides the functionality providing backup assistance,the rear-view camera 14 can offer additional lane sensing measurements;the vehicle orientation with respect to the tangent of the lane θ_(T) atthe rear, and the rear lateral displacement Δy_(T) at the rear bumper,where the distance d_(T) is defined as the distance from the center ofgravity point 56 to the rear bumper of the host vehicle 50. The twoadditional camera measurements, θ_(T) and Δy_(T), are valuable in designof a robust fusion system for lane sensing. They are especially usefulin inclement weather and lighting conditions, such as front low-anglesun, partially snow-covered lane markers, reduced visibility due to fog,and the like, where the quality of images from the forward-view camera12 would be reduced.

FIG. 6 is a block diagram of a first embodiment of a lane fusion system240 using input from two cameras. In the system 240, a full-fledgedforward lane sensor system 242 and a full-fledged rear lane sensorsystem 244 each include a camera and a processor, and can detect andtrack lane boundaries at their respective ends of the host vehicle 50.The forward lane sensor system 242 and the rear lane sensor system 244provide their measurements to a lane fusion module 246 which computesenhanced lane boundary and orientation information. The forward lanesensor system 242 sends measurements θ_(F), Δy_(F), and κ to the fusionmodule 246 at a fixed sample rate (e.g., 10 Hz). The rear lane sensorsystem 244 sends measurements θ_(T) and Δy_(T) at the same fixed samplerate. The forward lane sensor system 242, the rear lane sensor system244, and the fusion module 246 are interconnected by a serial network248, which may use the Control Area Network (CAN) or other protocol.

The fusion module 246 takes inputs from both the front and rear lanesensor systems 242 and 244, and vehicle dynamic sensors 250, and outputsthe enhanced lane information: vehicle orientation with respect to thetangent of the lane (θ), displacement of the front bumper center to thelane boundaries (Δy), and the lane curvature (κ). As mentionedpreviously, the lane information could be used by various downstreamapplications.

Let the measurements from the vehicle dynamic sensors 250 includevehicle speed (v_(H)) and yaw rate (ω_(H)). Then the following Kalmanfilter is designed to fuse the information from both the front and rearlane sensor systems 242 and 244.

Let the state variables be s=(κ, θ, Δy, φ_(F), φ_(T)), where κ, θ and Δyare defined as above; and φ_(F) and φ_(T) are the azimuth misalignmentof the front and rear lane sensor systems 242 and 244, respectively.

The state dynamic equation is written as:κ′=κ+v _(κ)θ′=θ−ω_(H) ΔT+κv _(H) ΔT+v _(θ)Δy′=Δy+v _(H) ΔTθ+v _(Δy)φ′_(F)=φ_(F)φ′_(R)=φ_(R)  (16)or in short as:s′=Fs+u+Gv  (17)where v=(v_(κ), v_(θ), v_(Δy))^(T) denotes a zero-mean Gaussian whitenoise vector modeling the uncertainty of the state dynamics model;

${F = \begin{bmatrix}1 & 0 & 0 & 0 & 0 \\{v_{H}\Delta\; T} & 1 & 0 & 0 & 0 \\0 & {v_{H}\Delta\; T} & 1 & 0 & 0 \\0 & 0 & 0 & 1 & 0 \\0 & 0 & 0 & 0 & 1\end{bmatrix}},{u = \begin{bmatrix}0 & {{- \omega_{H}}\Delta\; T} & 0 & 0 & 0\end{bmatrix}^{T}},{{{and}\mspace{14mu} G} = {\begin{bmatrix}1 & 0 & 0 & 0 & 0 \\0 & 1 & 0 & 0 & 0 \\0 & 0 & 1 & 0 & 0 \\0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0\end{bmatrix}.}}$

The measurement model can be written as:θ_(F)θ+φ_(F) +w _(θ) _(F)Δy _(F) =Δy+w _(Δy) _(F)κ_(F) =κ+w _(κ)θ_(T)=θ+φ_(T) +w _(θ) _(T)Δy _(T) =Δy+w _(Δy) _(t)   (18)or in short as:

$\begin{matrix}{{o = {{Hs} + w}}{{{{where}\mspace{14mu} H} = \begin{bmatrix}0 & 1 & 0 & 1 & 0 \\0 & 0 & 1 & 0 & 0 \\1 & 0 & 0 & 0 & 0 \\0 & 1 & 0 & 0 & 1 \\0 & 0 & 1 & 0 & 0\end{bmatrix}},}} & (19)\end{matrix}$o=[θ_(F) Δy_(F) κ_(F) θ_(T) Δy_(T)]^(T), and w=[w_(θ) _(F) w_(Δy) _(F)w_(κ) w_(θ) _(T) w_(Δy) _(T) ]^(T) is a zero-mean Gaussian white noisevector modeling the quality of the measurements from the front and rearlane sensor systems 242 and 244.

In summary, the following Kalman filtering procedure jointly estimatesthe misalignment angles and the lane parameters:

-   -   1) Randomly choose small numbers to initialize the misalignment        parameters φ_(F)(0) and φ_(T)(0); combining the misalignment        parameters with the first measurement from the front lane sensor        242 yields s(0)=(κ_(F)(0), θ_(F)(0), Δy_(F)(0), φ_(F)(0),        φ_(T),(0))^(T), and a covariance matrix P(0) is chosen for s(0).    -   2) When the new measurement at time instant t arrives, the        previous state vector is written as s(t−1); the predicted state        at time instant t can be written as {tilde over        (s)}(t)=Fs(t−1)+u(t), and the covariance matrix {tilde over        (P)}(t)=P(t−1)+GQG^(T), where Q is the covariance matrix of the        noise vector v.    -   3) Let the measurement at time instant t be o; thus the updated        state vector at time instant t is:        e=o−h({tilde over (s)}(t))        S=H{tilde over (P)}(t)H ^(T) +R        K={tilde over (P)}(t)H ^(T) S ⁻¹        ŝ(t)={tilde over (s)}(t)+Ke        P(t)=(I−KH _(t)){tilde over (P)}(t)        where R is the covariance matrix.    -   4) Output ŝ(t) as the fusion output.    -   5) Go to Step 2.

Using the above procedure, the fusion module 246 of the system 240computes a combined set of lane parameters for the host vehicle 50,while simultaneously determining the misalignment parameters for thefront and rear lane sensor systems 242 and 244.

FIG. 7 is a block diagram of a second embodiment of a lane fusion system300 using input from two cameras. The system 300 does not includefull-fledged lane sensor systems at the front and rear. Instead, thesystem 300 includes a forward-view camera 302 and a rear-view camera304. The cameras 302 and 304 only capture images and send them to afusion module 320, which combines both images together, detects andtracks the lane markers.

Images from the forward-view and rear-view cameras 302 and 304,respectively, are provided to box 306 to find local high intensityregions. The key idea of the box 306 is to find stable localhigh-intensity regions in different spatial scales. The algorithm beginswith building a Gaussian pyramid. At each pyramid scale, the image issubtracted by the enlarged coarse-level image, which is further blurred.Then the local maximum finding operation is applied to the differenceimages at different scales, and all maxima whose height is less than athreshold h are suppressed. Thus the binary images of possible lanemarkers are derived at the box 306.

At box 308, the detected pixels of curbs and stripes are projected ontoground plane in the vehicle coordinate system based on the cameracalibration parameters. At box 310, point clouds of the projected pixelsfrom the box 308 are first clustered based on the similarity measure(distance). Close pixels are clustered into a single component. Then thecomponents are classified based on their geometry shape. Componentswhose shape matches with curbs and lane stripes are selected, and thenline fitting and arc fitting procedures are applied to fit the stripecandidates. The components whose shape do not match with a line or anarc are discarded.

At box 312, the fitted stripes in vehicle coordinate system are thenlinked into a lane boundary. At box 314, lane information is tracked andoutput. This includes: monitoring the fitted stripes and data fromvehicle dynamic sensors; tracking the lane boundary; and estimating laneinformation, including the lane curvature (κ), the vehicle orientationwith respect to the tangent of the lane (θ), and the displacement of thefront bumper center to the lane boundaries (Δy). Details of thealgorithms used in the boxes 308-314 are given below.

The projection algorithm of the box 308 requires the following cameraintrinsic parameters:

-   -   Focal length: The focal length in pixels, [f_(u), f_(v)];    -   Optical center: [c_(u), c_(v)];    -   Skew coefficient: The skew coefficient defining the angle        between the x and y pixel axes is stored in the scalar α_(c);    -   Distortions: The image distortion coefficients (radial and        tangential distortions) are stored in the vector k_(c)=(k₁, k₂,        k₃, k₄, p₁, p₂), where (k₁, k₂, k₃, k₄) is radial distortion        coefficients and (p₁, p₂) is tangential coefficients;        and camera extrinsic parameters:    -   Translation vector T;    -   Rotation matrix R;        The camera extrinsic parameters are estimated through a camera        calibration process, many of which are known in the art, and        which need not be discussed here.

An iterative procedure used to remove the distortion is outlined below.The input includes a set of pixels S={(u_(i), v_(i))|i=1, . . . , N} andthe above-defined camera intrinsic parameters. The output is therectified set of pixels S′={(u′_(i), v′_(i))|i=1, . . . , N}. Theprocedure is as follows:

-   -   1) For each pixel s_(i)=(u_(i), v_(i)), i=1, . . . N;    -   2) Iteratively execute the following steps 20 times:        -   a. Let

$u = \begin{bmatrix}u_{i} \\v_{i}\end{bmatrix}$

-   -   and r=∥x∥.        -   b. Compute radial correction:            k _(rad)=1+k ₁ r+k ₂ r ² +k ₃ r ³ +k ₄ r ⁴.        -   c. Compute tangential correction:

${\Delta\; u} = {\begin{bmatrix}{{2p_{1}u_{i}v_{i}} + {p_{2}\left( {r^{2} + {2u_{i}^{2}}} \right)}} \\{{p_{1}\left( {r^{2} + {2v_{i}^{2}}} \right)} + {2p_{2}u_{i}v_{i}}}\end{bmatrix}.}$

-   -   -   d. Correct the pixel u=(u+Δu)/k_(rad).

    -   3) Output u as the final corrected pixel (u′_(i), v′_(i)).

After the above rectification, or distortion removing process, thefollowing transformation can be applied. The input includes a set ofrectified pixels S′={(u′_(i), v′_(i))|i=1, . . . , N} and the cameraextrinsic parameters described above. The output is the detected lanemarker points projected onto the vehicle frame: X={(x_(i), y_(i))|i=1, .. . N}. The transformation process is as follows:

-   -   1) For each pixel s_(i)=(u_(i), v_(i)), i=1, . . . , N;

${{a.\mspace{14mu}{Let}}\mspace{14mu} u} = {{\begin{bmatrix}u_{i}^{\prime} \\v_{i}^{\prime}\end{bmatrix}\mspace{14mu}{and}\mspace{14mu} K_{K}} = {\begin{bmatrix}f_{u} & {\alpha_{c}f_{u}} & c_{u} \\0 & f_{v} & c_{v} \\0 & 0 & 1\end{bmatrix}.}}$

-   -   -   b. Compute P=K_(k)[R T].        -   c. Let H=[p₁ p₂ p₄] where p_(j), j=1, . . . , 4 is the            column vector.        -   d. Compute z=H⁻¹u.

    -   2) Output z as the projected pixel (x_(i), y_(i)) in ground        plane in vehicle frame.

The above rectification and transformation procedures are applied at thebox 308 to provide a set of highlighted pixels, that is, points that arecandidate curb or lane stripe points, in the vehicle coordinate frame.Then, at the box 310, the pixels or points are clustered together intocurbs and lane stripes. Given the lane marker pixel set

$X = \left\{ {{\left. z_{i} \middle| z_{i} \right. = \begin{bmatrix}x_{i} \\y_{i}\end{bmatrix}},} \right.$i=1, . . . N}, the pixels are first clustered into stripes and then thestripes are fit into line or arc segments.

First, in order to cluster adjacent pixels into a stripe, a similaritygraph G=(V, E) is constructed where the vertex set is defined as thepixels on ground, i.e., V={z_(i)|i=1, . . . N} and the edge set E isdefined as a set of pixel pairs if each pixel pair's distance on theground plane is less than a threshold (T_(sep)), or each pixel pair isin 8-neighbor proximity of each other in the image plane, i.e.,E={(z_(i),z_(j))|∥z_(i)−z_(j)∥<T_(sep) v Neighbor(s_(i),s_(j))}, wheres_(i) and s_(j) are the corresponding locations in the image plane; andNeighbor(s_(i),s_(j)) is true if s_(i) and s_(j) are in 8-neighborproximity of each other. In this clustering methodology, 8-neighborproximity means that a second pixel is one of the 8 nearest neighbors(immediately left, right, above, below, above-left, above-right,below-left, or below-right) of a first pixel, in an approximatelyrectangular grid of pixels.

Next a depth-first search (DFS) strategy is applied to partition thegraph into connected components: {X₁, . . . , X_(c)}. Then each of theclustered stripes is fitted with a line or an arc.

Let z_(i)=(x_(i), y_(i)), i=1 . . . , N_(c) be a pixel in a detectedstripe. The stripe can be fitted by a line parametric equation (Ax+By=d,such that A²+B²=1). The parameters A, B and d can be estimated vialeast-squares, such as minimizing:

$\begin{matrix}{{{D\;\beta}}^{2},{D = \begin{pmatrix}x_{1} & y_{2} & 1 \\x_{2} & y_{2} & 1 \\\vdots & \vdots & \vdots \\x_{N_{c}} & y_{N_{c}} & 1\end{pmatrix}},{\beta = \begin{pmatrix}A \\B \\d\end{pmatrix}}} & (20)\end{matrix}$which can be solved by finding the eigenvector of X with the smallesteigenvalue λ_(m):Dβ=λ _(m)β  (21)The fitting residue is defined as e=λ_(m).

The width W and length L of the stripe are computed as:

$\begin{matrix}{{W = {{\max\limits_{i}\left( {z_{i}^{T}n} \right)} - {\min\limits_{i}\left( {z_{i}^{T}n} \right)}}},{L = {{\max\limits_{i}\left( {z_{i}^{T}t} \right)} - {\min\limits_{i}\left( z_{i}^{T} \right)}}}} & (22)\end{matrix}$respectively, where n and t are the normal and tangential vectors (unitlength) of the line segment, i.e.,

$n = {{\begin{bmatrix}\frac{A}{r} & \frac{B}{r}\end{bmatrix}\mspace{14mu}{and}\mspace{14mu} d^{\prime}} = {{\frac{d}{r}\mspace{14mu}{with}\mspace{14mu} r} = {\sqrt{A^{2} + B^{2}}.}}}$Then t is derived by rotating n by 90 degrees.

The two endpoints of the stripe are:

$\begin{matrix}{{e_{s} = {z_{m} - {\left( {{n^{T}z_{m}} - d} \right)^{\prime}n}}}{e_{e} = {z_{M} - {\left( {{n^{T}z_{M}} - d} \right)^{\prime}n}}}{{{where}\mspace{14mu}{indices}\mspace{14mu} m} = {{\underset{{i = 1},\ldots\mspace{14mu},N_{c}}{\arg\;\min}\left( {z_{i}^{T}t} \right)\mspace{14mu}{and}\mspace{14mu} M} = {\underset{i,1,\ldots\mspace{14mu},N_{c}}{\arg\;\max}{\left( {z_{i}^{T}t} \right).}}}}} & (23)\end{matrix}$

The orientation (angle) of the stripe is φ=a tan 2 (A, B).

If the residue of line fitting is larger than a threshold, the stripe isfit again using a circle parametric equation (x²+y²+a₁x+a₂y+a₃=0). Theparameters a₁, a₂, and a₃ can be estimated via least-squares, such asminimizing:

$\begin{matrix}{{{{C\;\alpha} - b}}^{2},{C = \begin{bmatrix}x_{1} & y_{1} & 1 \\x_{2} & y_{2} & 1 \\\vdots & \vdots & \vdots \\x_{N_{c}} & y_{N_{c}} & 1\end{bmatrix}},{b = \begin{bmatrix}{- \left( {x_{1}^{2} + y_{1}^{2}} \right)} \\{- \left( {x_{2}^{2} + y_{2}^{2}} \right)} \\\vdots \\{- \left( {x_{N_{c}}^{2} + y_{N_{c}}^{2}} \right)}\end{bmatrix}},{\alpha = \begin{bmatrix}a_{1} \\a_{2} \\a_{3}\end{bmatrix}}} & (24)\end{matrix}$with respect to α.

The solution of the above least squares is α=(C^(T)C)⁻¹C^(T)b. Theradius and center of the fitted circle can be written as:

$\begin{matrix}{{R = {{\left( {a_{1}^{2} + a_{2}^{2}} \right)/4} - a_{3}}}{x_{c} = {- \frac{a_{1}}{2}}}{y_{c} = {- \frac{a_{2}}{2}}}} & (25)\end{matrix}$respectively.

The two endpoints of the fitted arc can be computed as:e _(s) =[x _(c) +R cos φ_(m) y _(c) +R sin φ_(m)]^(T)e _(e) =[x _(c) +R cos φ_(M) y _(c) +R sin φ_(M)]^(T)  (26)and the stripe's orientation (angle) at the endpoints are φ_(s)=φ_(m)and φ_(e)=φ_(M), where the indices

$m = {\underset{{i = 1},\ldots\;,N_{c}}{\arg\;\min}\left( {{atan}\left( {{y_{i} - y_{c}},{x_{i} - x_{c}}} \right)} \right)}$and$M = {\underset{{i = 1},\ldots,N_{c}}{\arg\;\max}{\left( {{atan}\left( {{y_{i} - y_{c}},{x_{i} - x_{c}}} \right)} \right).}}$

The width W and length L of the stripe are computed as follows:W=max(∥z _(i) −c∥)−min(∥z _(i) −c∥)  (27)andL=∥e _(s) −e _(e)∥  (28)respectively, where c=[x_(c) y_(c)]^(T) denotes the center of thecircle.

In summary, the output of the box 310 is a list of stripes fitted withline segments with the following parameters; normal vector (n), distanceto the origin (d′), width (W), length (L), orientation (φ), and startpoints (e_(s)); or arc segments with the following parameters; thecenter of the circle (c), radius (R), width (W), length (L), and two endpoints' positions (e_(s) and e_(e)).

FIG. 8 is a diagram 400 which shows an example of lane striperepresentation for a scenario where the following have been detected:line segment #1 represented by end point 402 and normal vector 502, linesegment #2 (404, 504), line segment #3 (414, 514), and arc segment withradius 420, center (c) 422, first end point 406, and second end point412. The following steps are used at the box 312 to link the stripesinto left and right lane boundaries.

First, any stripe whose aspect ratio (L/W) is less than a threshold isremoved. Only slim stripes are kept for further processing. Then longarc or line segments are broken down into short segments, and eachsegment is represented by a start end point (e) and a tangential vector(t). For example, in the diagram 400, the start end point and tangentialvector for line segment #1 are represented as (402, 602); and the longarc is broken up into four end points: (406, 606), (408, 608), (410,610), and (412, 612).

To estimate the overall lane geometry information at the box 314 (i.e.,the lane curvature κ, the vehicle's orientation with respect to the lanetangent θ, and the displacement to lane boundaries Δy), an estimate isneeded for the position of center c.

Given a set of stripe segments {(e_(k),t_(k))|k=1, . . . , K}. For eachsegment, (e_(k),t_(k)) there is a normal (dashed lines in the diagram400) passing through c, i.e., t_(k) ^(T)(c−e_(k))=0. Let t_(k)=(t_(xk),t_(yk)). Therefore, finding c is equivalent to minimizing the followingleast squares:

$\begin{matrix}{{{{Ec} - \gamma}},{E = \begin{bmatrix}t_{x\; 1} & t_{y\; 1} \\t_{x\; 2} & t_{x\; 2} \\\vdots & \vdots \\t_{xK} & t_{y\; K}\end{bmatrix}},{\gamma = \begin{bmatrix}{t_{1}^{T}e_{1}} \\{t_{2}^{T}e_{2}} \\\vdots \\{t_{K}^{T}e_{K}}\end{bmatrix}}} & (29)\end{matrix}$The solution of the above least squares is c=(E^(T)E)⁻¹E^(T)γ. Thecurvature of the lane can be written as:

$\begin{matrix}{\kappa = \left\{ \begin{matrix}\frac{1}{c} & {{if}\mspace{14mu} c{\mspace{11mu}\;}{is}\mspace{14mu}{on}\mspace{14mu}{left}\mspace{14mu}{side}} \\{- \frac{1}{c}} & {{otherwise}.}\end{matrix} \right.} & (30)\end{matrix}$

The vehicle's orientation with respect to the lane tangent can becomputed as:θ=a tan 2(c _(x) ,c _(y))  (31)where c_(x) is shown as dimension 426 and c_(y) is shown as dimension428 in the diagram 400.

FIG. 9 is a histogram 700 which shows an example of how the displacementto the lane boundaries can be computed. Let {z_(j)|j=1, . . . , M}denote the pixels of detected lane stripes. The histogram 700 isconstructed which graphs the distance to the center c for all of thesepixels (i.e., d_(j)=∥z_(j)−c∥, j=1, . . . , M). The histogram 700 has anorigin point 702.

Displacement to left lane boundary y_(L) is distance 704 from the originpoint 702 to the left local peak in the histogram 700, whiledisplacement to right lane boundary y_(R) is distance 706 from theorigin point 702 to the right local peak.

Equations (29)-(31) estimate the lane using data from a single framefrom the cameras 302 and 304. The method can be extended to includetracking and data from vehicle dynamic sensors. Two such methods areproposed. For both methods, the state variables are defined as s=(κ, θ,Δy) where the variables are defined as the lane curvature (κ), vehicleorientation with respect to the tangent of the lane (θ), anddisplacements to the lane boundaries (Δy), respectively. Let the vehiclespeed (v_(H)) and yaw rate (ω_(H)) denote measurements from the vehicledynamic sensors.

For the first method, a Kalman tracking procedure is used to estimatethe lane parameters. FIG. 10 is a flow chart diagram 800 of the Kalmantracking method. The steps are as follows:

-   -   1) At box 802, initialize the state vector s(0) with first        measurement from the system 300 (Equations (29)-(31)), and        choose a covariance matrix P(0) for s(0).    -   2) Wait at decision diamond 804 for new data to arrive; when the        new measurement at time instant t arrives, at box 806 write the        previous state vector as s(t−1); then at box 808 the predicted        state s(t) at time instant t can be written as:        κ′=κ        θ′=θ−ω_(H) ΔT+κv _(H) ΔT        Δy′=Δy+v _(H) ΔTθ    -    where ΔT is the time increment, and the projected state vector        s′(t)=[κ′ θ′ Δy′].    -   3) Also at the box 808, the circle center is computed as:

$c^{\prime} = {\begin{bmatrix}{\frac{1}{\kappa^{\prime}}\sin\;\theta^{\prime}} & {\frac{1}{\kappa^{\prime}}\cos\;\theta^{\prime}}\end{bmatrix}.}$

-   -   4) At box 810, the detected stripes (e_(k),t_(k)) from the        cameras 302 and 304 are provided; then at box 812, a gating        operation is performed to identify outliers of the detected        stripes, using the following criteria:

$\frac{{\left( {e_{k} - c^{\prime}} \right){{}_{}^{}{}_{}^{}}}}{{e_{k} - c^{\prime}}} < T$

-   -    where T is a threshold value; a stripe will be treated as an        outlier if the above is not true.    -   5) At box 814, compute the current lane geometry information;        for all stripes remaining after the gating of the box 812, the        least squares are minimized using Equation (29) to find the        solution for the updated center ĉ; then κ_(m) and θ_(m) are        computed through Equations (30)-(31), respectively, and        displacements Δy_(m) through building the histogram.    -   6) At box 816, perform a measurement correction; treat κ_(m),        θ_(m), and Δy_(m) as the direct measurement of the state        variables; the following measurement equations can be written:        θ_(m) =θ+w _(θ) _(m)        Δy _(m) =Δy+w _(Δy) _(m)        κ_(m) =κ+w _(κ) _(m)    -    where (w_(θ) _(m) w_(Δy) _(m) w_(κ) _(m) )^(T) is a zero-mean        white Gaussian noise vector, whose covariance matrix is a        function of the residue in the least squares minimization of        Equation (29); then a Kalman filter is applied to the obtain the        final output s(t) and the corresponding covariance matrix P(t).    -   7) At box 818, output the updated lane geometry information, and        go back to the decision diamond 804.

The Kalman tracking procedure described above and on the flow chartdiagram 800 represents the first method for computing lane curvature andvehicle orientation information, using images from the forward-view andrear-view cameras 302 and 304 and data from vehicle dynamics sensors.The second method uses a particle filter. FIG. 11 is a flow chartdiagram 900 showing the particle filter method, which uses the followingsteps to compute the lane parameters:

-   -   1) At box 902, initialize the state vector s(0) with a set of        particles (random sample of geometry information): {(s_(i)(0),        w_(i))|i=1, . . . , M} and the weight

${w_{i} = \frac{1}{M}},$for i=1, . . . , M.

-   -   2) Wait at decision diamond 904 for new data to arrive; when the        new measurement data at time instant t arrives, for each of the        particles, κ_(m), θ_(m) and Δy_(m) are calculated using the        steps 2) to 5) of the Kalman tracker; that is;        -   a. At box 906, write the previous state vector as s(t−1).        -   b. At box 908, calculate the predicted state s(t); also            compute the circle center c′.        -   c. At box 910, provide detected stripes from both cameras;            at box 912, perform a gating operation to identify outlier            stripes.        -   d. At box 914, compute the current lane geometry information            using Equations (29)-(31) and the histogram.    -   3) Then the value of the i-th particle becomes s′_(i)(t)=(κ_(m),        θ_(m), Δy_(m)); let Δ_(i) denote the residue of the estimation        for the i-th particle; at box 916, compute the new weight of the        particle as

${w_{i}^{\prime} = {\exp\left( {- \frac{\Delta_{i}^{2}}{2\sigma}} \right)}},$where σ is a predefined constant.

-   -   4) At box 918, compute the weighted average of the particle set        ŝ(t) as:        ŝ(t)=Σ_(i=1) ^(M) s _(i)(t)w _(i)/Σ_(i=1) ^(M) w _(i)    -    and output ŝ(t).    -   5) At box 920, apply importance re-sampling, a standard        statistical procedure, to the updated particle set as        {(s′_(i)(t), w′_(i))|i=1, . . . , M}; this yields a set of        random samples of the updated lane geometry information at box        922.    -   6) Go to Step 2, the decision diamond 904.

As described above and shown on the flow chart diagrams 800 and 900,either the Kalman filter method or the particle filter method can beused to compute the lane geometry information—the lane curvature κ, thevehicle's orientation with respect to the lane tangent θ, and thedisplacement to lane boundaries Δy—using images from the forward-viewand rear-view cameras 302 and 304, and vehicle dynamics sensors, asinput. The resultant lane geometry information can be used by downstreamapplications, such as a lane departure warning system.

The methods and systems disclosed herein, by using the image dataavailable from a rear-view camera, and combining it with image data froma forward-view camera and other sensors, provide more robust capabilityfor lane sensing or lateral control. The two-camera system not onlymakes use of more input data under normal conditions, but also providesa usable source of image data to allow operation of the system whenconditions are unfavorable for forward-view imaging. Vehiclemanufacturers and consumers can benefit from these systems, which takeadvantage of existing rear-view imaging capability in many vehicles tooffer improved system performance and reliability, while incurring nonew hardware-related costs.

The foregoing discussion discloses and describes merely exemplaryembodiments of the present invention. One skilled in the art willreadily recognize from such discussion and from the accompanyingdrawings and claims that various changes, modifications and variationscan be made therein without departing from the spirit and scope of theinvention as defined in the following claims.

What is claimed is:
 1. A method for determining a vehicle's position ina lane, said method comprising: providing data from a forward-viewcamera module onboard the vehicle; providing data from a rear-viewcamera module onboard the vehicle; providing data from vehicle dynamicssensors onboard the vehicle; and fusing the data from the forward-viewcamera module, the rear-view camera module, and the vehicle dynamicssensors into a combined set of lane geometry information to determinethe vehicle's position in the lane, including using a Kalman filterroutine to jointly estimate the lane geometry information andmisalignment angles of the forward-view camera module and the rear-viewcamera module, where the Kalman filter routine includes lane curvature,vehicle tangent angle to the lane, front bumper displacement to a laneboundary, and azimuth misalignment angles of the forward-view cameramodule and the rear-view camera module as state vector variables andsimultaneously solves for all of the state vector variables.
 2. Themethod of claim 1 wherein the lane geometry information includes lanecurvature, vehicle orientation with respect to a tangent to the lane,and displacements to left and right lane boundaries.
 3. The method ofclaim 1 wherein providing data from a forward-view camera moduleincludes providing front lane geometry information from a frontfull-fledged lane sensing system, and providing data from a rear-viewcamera module includes providing rear lane geometry information from arear full-fledged lane sensing system.
 4. The method of claim 1 whereinproviding data from a forward-view camera module includes providingforward image data from a front camera, and providing data from arear-view camera module includes providing rear image data from a rearcamera.
 5. The method of claim 4 wherein fusing the data from theforward-view camera module, the rear-view camera module, and the vehicledynamics sensors includes finding high intensity pixels in the forwardimage data and the rear image data, projecting the high intensity pixelsonto a vehicle coordinate frame, detecting curbs and lane stripes fromthe high intensity pixels, linking the curbs and lane stripes into thelane, and tracking the lane geometry information.
 6. The method of claim4 wherein fusing the data from the forward-view camera module, therear-view camera module, and the vehicle dynamics sensors includes usinga Kalman filter.
 7. The method of claim 4 wherein fusing the data fromthe forward-view camera module, the rear-view camera module, and thevehicle dynamics sensors includes using a particle filter.
 8. The methodof claim 5 wherein finding high intensity pixels in the forward imagedata and the rear image data includes using a Gaussian pyramid routine,where binary images at different spatial scales are subtracted from eachother.
 9. The method of claim 5 wherein projecting the high intensitypixels onto a vehicle coordinate frame includes using an iterativeprocedure to remove radial and tangential distortion, and a rotationaland translational transformation procedure, to produce a plurality ofhigh intensity pixels in the vehicle coordinate frame.
 10. The method ofclaim 5 wherein detecting curbs and lane stripes from the high intensitypixels includes using a similarity graph and depth-first search routine,which clusters neighboring high intensity pixels into stripes.
 11. Themethod of claim 10 wherein detecting curbs and lane stripes from thehigh intensity pixels also includes fitting the stripes with a linesegment or an arc using a least-squares routine.
 12. The method of claim5 wherein linking the curbs and lane stripes into the lane includesfinding a center of curvature of the lane using a least-squares routineapplied to the curbs and lane stripes.
 13. The method of claim 12wherein tracking the lane geometry information includes computingdisplacements to left and right lane boundaries using a histogramroutine, and computing lane curvature and vehicle orientation withrespect to a tangent to the lane using the center of curvature.
 14. Amethod for determining a vehicle's position in a lane, said methodcomprising: providing image data from a forward-view camera onboard thevehicle; providing image data from a rear-view camera onboard thevehicle; and fusing the image data from the forward-view camera and therear-view camera into a combined set of lane geometry information todetermine the vehicle's position in the lane, including using a Kalmanfilter routine to jointly estimate the lane geometry information andmisalignment angles of the forward-view camera and the rear-view camera,where the Kalman filter routine includes lane curvature, vehicle tangentangle to the lane, front bumper displacement to a lane boundary, andazimuth misalignment angles of the forward-view camera and the rear-viewcamera as state vector variables and simultaneously solves for all ofthe state vector variables.
 15. The method of claim 14 wherein fusingthe data from the forward-view camera and the rear-view camera includesfinding high intensity pixels in the image data, projecting the highintensity pixels onto a vehicle coordinate frame, detecting curbs andlane stripes from the high intensity pixels, linking the curbs and lanestripes into the lane, and tracking the lane geometry information. 16.The method of claim 14 wherein the lane geometry information includeslane curvature, vehicle orientation with respect to a tangent to thelane, and displacements to left and right lane boundaries.
 17. A systemfor determining a vehicle's position in a lane, said system comprising:a first camera for capturing images of a forward view from the vehicle;a second camera for capturing images of a rear view from the vehicle;and a processor configured to receive the images from the first andsecond cameras and compute lane geometry information using a fusion ofthe images from the first and second cameras, including using a Kalmanfilter routine to jointly estimate the lane geometry information andmisalignment angles of the first camera and the second camera, where theKalman filter routine includes lane curvature, vehicle tangent angle tothe lane, front bumper displacement to a lane boundary, and azimuthmisalignment angles of the first camera and the second camera as statevector variables and simultaneously solves for all of the state vectorvariables.
 18. The system of claim 17 wherein the processor finds highintensity pixels in the images from the first and second cameras,projects the high intensity pixels onto a vehicle coordinate frame,detects curbs and lane stripes from the high intensity pixels, links thecurbs and lane stripes into the lane, and tracks lane geometryinformation.
 19. The method of claim 17 wherein the lane geometryinformation includes lane curvature, vehicle orientation with respect toa tangent to the lane, and displacements to left and right laneboundaries.